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ABSTEfACT , ^ ' ' ' ' . \ 

This report provides a sunmary record of the 
development and evaluation of a training package product that 
includes three separate sets^'of training materials developed^ to ' 
increase the skille knowledge, and understanding of school personnel 
in the area of instructional planning and management. Instructional 
planning is defined generally as that area' ot educational 
administration concerned with providing j^duc^tional opportunities for 
children through the establishment of instructional, program purposes, 
the design and implementation of programs, and t^e evaluation of 
instruct'ipnal prqgrams.^ This packag'e is directed toward establishing 
instr^uctional program purposes in three units— setting goals, 
analyzing problems, and deriving objectives, ^adh unit is described, 
it:s development is delineated, and the evaluation results are 
announced. (Author/DH) ' . \^ ' 
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- I Thi? report ^is intended to provide a surffnary* record q^f ' the. development 

and evaluation of one ,o£ the products produced by the Educaticn§l Manageuient ' 

\" " * ' / • ^ . * • * 

Program* ,This product is a^ery major one., consisting o£ tlii;ee separate ^ . 

training units; it was also the first in what was originally, planned as a - 

complete, responsive system 'directed to improvijig.gx^hool .instryictional planning 

and management. Jteferenc^ ,is made in the report to^,a nuiii)er of inbi;;e detailed 

.reports and memoranda which constitute the dodumeatation of Jhe development ^d 

evalu^ion processes. Because of the^ product conplexity and originality the\ 

documentation is^ qfiite \:oliifninous , and can not 'be. included with tfte report, AlX 

of the documents referenced are in program file^^, and most of them*have been 

submitted to various funding agencies that siq)ported thd wprjc over the last 

$ive years. Oriiy a .final evaluation' report c6n5)leted in Itecenber 1974 is in- 

eluded -as an attadiment to this report . . < ' ' 



the report has been written 'primarily for those ass o<^iated with th^ 
?fetiaial Institute of Education, and'possibly some .potential t^r§ of tke 
product, who need to make judgn^ents about product quality but don't have the 
time to becope familiar .in detail with the product itself and the-pxtensive 
reports prepared -during the developmeat_.and-evaluation^.pf the product* The 
lengthy history of development is described only in sufficient detail to suggest 
the amount of thought, care, and discipline ^that went into Creation of the units 
in tlie product. Evaluation information obtained during development are presen- 
ted, at only the most general level that would accurately reflect the grqss 
strengths and weaknesses of each unit. The filial independent e\/aluatioji of the 
units is quoted at greater length, since in jhe Wa'f analysis this is probably 
^ti\e report that should be given greatest weight. More than mosy reports in , 
educational development, the*report that follows is a sumAary of sumnaries* 



♦ c 



* , -v^ A'fevy r^6aders*m^y find the report valuable as routing dfevice to\locate 

, / critieal dpcunentat?.on, or to ^|assure .theiTi5.elves that the proce^dure^ were ^ ^ ** . 
docAnented.' They'^hoyld keep in miud that ph^ the crucial docunents ha\^^ 
— ^ ♦ teen^^ted. There ' are, niany* more vAiere thesgrcame from. . * * * ' - * ' . 
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^ \ , ** * * ' \ , ' * * . Program Director 
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' Jhe Developm^t -and 'Evaluation of 
The Determindng In&tYUGtlonal-Edlipose^tra 



. • . ... ■ f • ^ • 

TTie Dsterminijig Ihstructidnal Purpose?^ Training Package includes, three 

s.eparate sets of training materials developed to i.ncrease the, skill, Icnowledge 
^ ' ' * ' . ^ > ' ' 

' and understanding of school personnel in the area of instructional planning * . 
andimanageni^t. Instructional jilanning is defined generally as that area of i j 
educational administratis fthat is cmcemed with providing educational oj^or- 
tunities for children. This are'a iri^ be fiirtJier defined as involving^ the 

estabiishmfent of instnicti^al program purposes, the design and implementation 

I --^ . . , — — -.^^^^ v^^- ^ .... 

of prpgr'ams and tlfe%evaluation-of instructional. programs. The p^age is 
diTe/;ted to the first of these three functions; the'' three units are: Setting 
Goals, %ialy zing Problms, and Deriving Objectives. V ' " ' 

Intended Users ° - " " o' • 

_ *. L^.'Ihese.jnaterials^ are__|ntended for those school personnel who are directly 

concemed with the design or selection of school curricula (instructional pro^ 
grains) to be implemented in classrooms. The speqifiq:^ role titles of such staff 

will var)^ f,rqm district to district; but most often will include building!,^ . 

' . V * • • * ■' • ^ ^ \ 

principals, department heads, district curi'iculunf specialists , and teachers 

who are sen'ihg on curriculum pommi^ttees. ^ Depenxiing on the size and*organiza- 

. ^ i 

tion of a school district or system, the package materials may also be fomd^ 

useful by district office administrators other than thqse with direct curriculum 

responsibilities, Bd^fird of*Educatron mfembers , 'p^nts and studfelits. • It should 

be clear that the intended user gro.up can best be defined Ify function rather than* 

by a particular* role or title,' and that the group could inqlude staff nbt ' i 

^necessarily* falling under the heading of adminisitrators. / 
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Prpduct Purposes and Description 



* The ^oal of this package is. to assist -in the' preparation? of scho6r staff 

^ ' ^} ' y ^ ^> • . ^ . ^ 

in writing Tnorcjyseful statemejits o& goals .and [.obje^ctives for instructional ' ^ 
• , * * ; . ' , ^ *\ * ^« • 

, programs. The general objective^ for the package are to increase knowledge, 
con^rehension, and skills in the process of (a) selecting long- range^ goals and 
translating them into behaviorally stated goal indicators,, (b) anal)^zing in^ruc-;. 
tiohal problems to coupare their^importante' to the school or district and to 
(^termine where "the availlBle-resotifxies sljould be applied, and (c) deriving 
objectives from, jthe hi^ priority goals. ^Accon^ilishmeijt of these objectives 

, is judg^dVijiportant to .the design of programs and plaming for piogram evalua- , 
tign if scho9l.-s:|afff^are zp have confidence that the resurting programs will be 

•relevant to the scho^or district needs. Each of the ti^e units .that consti- 
tute the package is aifected to one of .these general dDjeptives. The units 
were, j;iowever, developed s.o that each could used by itself or in ^combi^iation 

— ' . r, X T " " ' ^* ~ 

with any-, other ^of 'the units, thtis enabling potential users to choose the. particu- 

lar unit(s) that seemed* mos.t J.ilcely to .meet the staff training needs they faced. 
• « • 

Egch unit is described separately belcw.. — \^ 

/ - ' . * 

Setting Goals . This mit consists of an Introduction and four integrated 

•a 

. training modules, and generally requires about twelve to fifteen hours of groi^) 

training time. There are no audio -visual materials, but auxiliary materials 

separate from the printed training booklets are used. The four modules aye: 

Deriving Mstrlct Goals, Refining Goals into GoalAjhdicators, Screening Sets of 

Goal ][ndicators, and Prioritizing Goal Indicators .\^In usuig the materials, each 

participant assumes the role of a member of a special planning, team in the "Mid J 

Cit5^ Unified School District." iVbrJcing m the team, the ^Jrticipants derive 

* • * *^ * 

» goals for the district, refine these into goal indicators which will meet speci- 

fied criteria, screen indicafdr.s , and establish priorities among the goal 



• 'I 

indicators which will be used for further planning, of the instructional^program. 

^l^ch of .the' training, in^the unit is accomplished through applica^on of defini- 
'' ' * . . • . . ' ' • » . 
tions and criteria for the 'goals and goal indicators to the goal establishment 

r— ' • ' . ■ £ 

prpblems .posed for the simulated school- district. . 

Analyzing Problems . Ihis Imit also consists entii^ly of pro^d materik. 
It is^njade of reading material, a workbook^ and auxiliary materi^, Thei^ ^ 
,is active participant involvement in completing the exercises in the workb^k.' 
The unit generally requires fifteen to twerity hours .b| ^\xp trying time, ^^"^ 
though most. often the upper limi't of'tiventy howrs wiU be. approadiedt if the. mit 
is be used as designed; 'There is an. Intyoducti-on and-sijc iiK)dules ,,itt.;the^iiiu^ 
the ^dules are identified in the £om of quesl^ions \hidi suggest the content of 
each inodule: Module 1 - Which of the pjcoblah signals received warrant further 
^consideratim?. Module 2 - What student outcome pVoblems do the signals inroly^? . 
Module 3 - What additional infonnation is Jieeded to analyze the pix)bleTis? Module 
4 " How should the information needed to analyze the problems be collected? 
Module 5 - Are the problems valid? McJdule 6 *- How serious are the prd)lems?. 

As with the first unit ^(Setting Goals) \ participants in the training -assume 

the role c^f a member of an instructional planning team in the simulated Mid City 

Uhified School District. ^The training is carried out thrpu^ the extensive 

application of definitions., explanations and criteria' to the simulated task of 

identifying an'd refining problem sisals, detemining arid collecting necessary 

additional infomation., and establishing priorities for responding to those S 
^ ^ ' ' . ' ' , Y 

problems that are real and serious. , • ^ ^ - 

Deriving Objectives , This unit, like the*oth6r two^ is^ conposed entirely 

ofVritten materials. Unlike the other two, however, .this unit includes pro- 

grammed learning^ materials as part of one of the modules,; and instead of using 

the simulated Mid City Uhified School District, the unit makes extensive use of 



quite artificial educational settings for presenting the problems to be dealt 
with. The purpose of. introducing artificial settings located on other planets 
is to facilitate an approach that is free of the conditions of ongoing^ school 
situations that might trigger Vstandard" approaches to the problems. Ten to 
*twelve nours are generally required to;Conplete the unit. 

The unit consists of four modules: Module 1 The 'Goal Refinement Process; 
Module 2 - Screening Objectives; Module 3 - Analyzing Sets of Curricular Objec- 
tives; Vhich is divided into the jprogranmed text referred tb above, covering 
, educational tajcdnomies, and an artificial simulation; and Ntodule^4 - District 
Planning for Goal Refinement, "^^This^unit places sanewhat greater stress on an 
understanding of^he processes involved, rather than on detailed definitions and 
criteria, to be applied, although the importance of the latter Xs not ignored. 

Coordinatoi^'^s Handbook. The Determining Instructional Purposes Package 
aljib incl udes .a C dordina:tor^s Handbook which contains material necessar)'' for. 



conducting the training using any of the three units. The'.material .in the 
,Handbodk is intended to be siifficient to permit a school ^stalf member to conduct 
the training, if he or 3he takes tlie time in advance to work through the units 
pn his or her own. In other words, no external consultant is needed to conduct 



the training, nor is any special training needed, provided the Coordinator will' 
take the time to read through the unit as part of the*prepaxation for offering 
the training. 

Development and Testihg ^ < . . 

, These three unlets as a group were Jlie first instructional planning and 
managanent training materials developed by the Laboratory. Consequently, a consid 
erable effort. was devoted to neods analysis, conceptualization and planning, and 
feasibility testing and replanning. The need for this kind^of training efvolved 
from work carried out in one of the components of sxi earlier program directed 



at increasing capability of school personnel to use and ^ply the outcand3 of 



educational research and development. The general specification^ for a train- 
ing unit "that would help school ^staffs identify; needs and deteimine goals and 
^objectives werfe first set forth in September 1969 (Far West Laboratory, 1969^, 
p. 80) ITie need for such a unit was based on ah extensive pilot study of a 
procedure to facilitate these puiposing functions (Carlisle, Cone, and Whitney). 

Additionally, reviews of the literature on .educational management and school • 

- - , - - O . * , - 

probrem solving i^ere carried en ccj^tinually. Conceptualization* and planning 

for the design of the proposed training unif were carried, out in 1970, and 

indicated that at least' three separate training units Would probably be> requir^ 

as specifications and unit scope were detaileH (Conmunication Program, January 

1970-; March 197QX. An analysis ^needs and system specifications from which 

the training, units could be derived was, <:on5)leted (Banathy and Jenks), and 

school staff were in^terviewedjextensively (Jenks; Jetiks and Gall). A conference 

of a variety of agencie^ involved in development of management training was also 

sponsored by the Educational Managment Program (Yory), The three. units an4 a. , . 

schedule for their development were formally identSfi^d (Far. West Laboratory, 

1970) and work started during the latter half of 1970. The Units m goal 

s • • >■ 

S'etting and problem ^alysis were started first, and were developed nearly 

<^ ' • . , _ • 

siimiltaneously. Feasibility and preliminary testing of these units was ccnple'ted 

(Gall and Walker) by the time that work on the unit dealing with objebtives 

wa$ initiated in January 19712 , * * 

» 

The main field test of all the units diad two major aims: ^ to test their 

* * ■* * % 

e'ffectiveness in achieving the intended .purposes , and to identify specific 
features of the'units that needed final fevision prior to publication. The 
three units were tested separately using non- Laboratory staff as coordinators 
of the training. TTie test ihstruments were designed to provide infomation on 



the jachieveinent .of' identified knowledge ahd skill objectives but the I'ai^e, 



nunfcer of q^stions were also directed to specific portions of moditles in eadi 
of the units in oyder to identify .necessary revi'SionsT . Participant opinions * 



*> 1 



about vari6us .aspects of the training activities ^ as well as the totaTunit were 

-obtained for all the units, but were cast, in terms specific to each mit„ - 

^ < • • • .^^^ 

^por^s on the evaluation plan desi^, instrumentation and results have been 

reported f^r the rest of each of the three units (Gall, Oakley and Walker; Gsill 

and York; Otto, Kalker, and Jenks). 'The results are sunmarized here in very. ' 

♦ . -\*^ 

brief foim. " . - , \ 

*^ ' ' 

Setting Goals/ The main ^eld test was started in October 1971 at fiva) 
sites with a total of" 74 trainees who completed the unit and provided data for 
analysis; ;tvA3 of the sites were in university classes and three were in school 
districts, implying predetermined decision standards based on both statistical 
and training significance, it was concluded that all six of the intended know- 



ledge or cognitive objectives and all three of the affective ol)jectives had 
b^en achieved; only one*^^^^ four skill objectives was achieved a'f^ the^^standard 
set, however, ^formation from the field test was used to revise the unit sa ' 
that two of the three ''failed" skill objectives wc/old more likely be achieved.* 
.The third of the "failed" objectives was eliminated from the uiiit so that no 
stated or implied claims for this objective wuld be made in the piilished foim^ 
of the unit. • , . ' 

Malyzirig Problems . The field test of this unit was conducted in the 

^^^^^^ - " 

period March to July 1971 using four different classes of students in a graduate 

educational administration course at San Francisco State liiiversity. The coor- 
dinators were three faculty numbers ^rom the university (the department head 
served as coordinator in two of the classes); 44 'trainees con^Jleted the unit 
and provided data for analysis, inlying the predetermined decision standards 



*to the data 'from the stnify led tp the following conclusions: six of *eight 

knowledge objectives, l'3 of 16 skill Qbje,ctivcs,.and,b£>tli of the major affective 

* ' ' " ' • ' ' , , ' ^* ' \ - • 

objectives had been* achieved^ Virtually ail of the. objectives that had not 

been acliioved were^^the focus .of a single module directed to the question 'How 
' . L ' 

good 1^ the information that iwas collected?'* It was concluded that the content 

of this-modul,d^ dealt with issues that were .too advanced relative to those 

• • " \ . . 

covered in the 6tHer, jnodulQs. Sfece , completion of this module was judged ^o be 

unnecessary to the successful' use 'of any of the others, the decision was made 

.to eliminate it ):omple^tely in^kny published fbim ^of the unit. ' ' , 

Deriving 03 j actives > .TWci versions ,of thi| unit were used in "field testing! 

During the development of the unit, the Anerican. Council on the Teaching of 

^^oreign- Language (ACTFL) prt^vided funds for the adaptation of the Deilving 

Objectives unit to. meet the needs of foreigi) language teachers. In this adapta- 

'tion, Nfodules 1 and 2 and the programned tfext fyom Module 3 were kept intact. 

Hie second hall of hbdule 3 was revised to incorporate problems ^levant to 

/ 

foreign language instruction, and Nfodule 4 was eliminated con^letely since ACTFL 
and program staff judged it not to be necessary for application of the outcones 
of the training in tiie actual school situation^. ' ^ 

The original unit was used Jn-^a^university course with 12 students, and 
two school district workshops^.with a t<>tal ^enrollmeat of 74; "^e modified unit 
was used in two workshops sponsored by ACTFL, one with 65 participants and thfe 
other with 168 participants. Following completiai of the field test, it was""., 
concluded that the unit had been successful in achieving l£Lof the ^5 knowledge 
and skill objectives when the predetermined standards were applied. The affec- 
tive evaluation items were grouped in termS of: (a) values of the training 
unit; (b) the jnsthodolog)' used in the training; and (c) the probable usefulri^s ' 
of the training. The uj;iit was, judged' to be successful in all three of these ^ 
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^^areat^. 'Only editorial and, fomating; revisions were made in -the preparation of 
copy for publication** 



Extefrri al Evaluation * ' . ' 

* «* • f ' ' * • > 

IVhen the content, -editorial, and format .revisions necessaiy as indicated by 
' - * . ■■ ' ^ ' 

jthe^f^eld tfests'had'beeij made', the progrm staff (Hood) requested and was granted 

' ' ■ • ■ ; ^ ■ ' " . • 

.apf»roVal to nake the DQtentiining Instructional Purposes Package available for 

• > ' tfn ' ^ - - 

sale by the Laboratory on an. interim basis.' ihe^ puipose of this interim* release 

^was to obtain infomation ^ott: the'^e'^^fectiveness of the"uni1s> in operational use 

situations for Aich it had be6n desig^ea, 'and to obtain informatioif about the 
.A , - .,v - 

•likely market f^;4 this mif, Final^copy for 'the entire unit was prepared j > 

printed By the Government Printing Office, and placed on sale by the Laboratory 

in the sunmer of 1973. The price. of pai .icipant materials for any one unit was ^ 

. \. \ ' , 

set at $8.95 per participant; the Qwrdinator's Handbook cost $4.50. A report 

on a ^)Teliminary n^arkfeting . study was prepared in 1972^ (Educational Managenent " 

. program, pp. 9^-132). ^TWo kinds of studjLes related to evaluation of the package 

as published ate stsifearifed^below-r^^^ • - ^ / 

Independent expert Veviev/ , In X972, Educational Testing Service^ (ETS]l * 

coridj(^ted a systematic review of conpleted educational deveiopmettt products for 

'the National Center Ipr E4;cational Conromication (NCEC) of the U.S.- Office of 

Edfication. For this review, development agencies across the nation were invited 



to siibnd't^conplfeted products to, be reviewe^'^.for, possible asyar^ of fimds to be 
us^4^in promotion of product distribution efforts, ^^ore than 100 products were 
submitted for reyiew. All of the products and available evaluation information 
about eac^i of them 'were ass.OTibled and abstracted by ETS. A' panel of ten spe.cial- 
ists caning frm'a variety of interests and disciplines (siQ)erinJ:endents,i^- 
teachers, schools of education. Boards of Mication, etc.) reviewed the products 
information, and recaransnded to NCEC those products which they believed 



merited the use of federal funds to encourage product dissemination • 

The DeteiTOxning Instructional Purposes Package was one of 23 products 
chosen for such reccamendation , although the dollar support recommendation was 

4 

fcontingent on corpletion of an independent evaluation study. The panel discussion 

summary included in the recommendation (Educational Testing Service)* said: ' ^ 

"Not only should an absolute evaluation be cajxied-out,,iut..a com- 
parative evaluation with competing products., notably ... Elementary 
School Evaluation Kit: Needs Assessment and ... Evaluation Woric- 
shop I: An Orientation, should be made. The product has the 
^ notable. and rare feature of showing means ^d confidence limits as 
well as gains and probable errors. It appears sonewhat more man- 
ageable than [the Elementary School Evaltiation Kit] . The evaluation 
of the product was good, but the criteria used coiild be more rigor- 
ously defined via independent cpmjjafative evaluation." 

Exterrfal Evaluation study . Late in 1973, partly in response to the recom- 

msndation'of t^ie ETS panel, planning was started for an independent evaluation 

of the package. In Februa^l974 a Request for Proposals (Far West Laboratory, 

1974) \iss s'dht to seven iiidividuals or agencies \ho were thought to be qualified 

and might gossibly be interested in submitting a proposal; one of these seven* 

forwarded the RFP to an eighth agency. Three proposals v;ere received and svb- 

mitted for review anid recomnaidation to two evaluation specialists not ^associated 

with the Far West Laboratory and three Laboratory staff menbers not associated 

with th6 Educational Management Program. This review process was described in 

the RFP, and was conducted as described. The recommendations were not unanimous 

for any of the proposals, althou^ they did tena to favor die of the contractors; 

one proposal v/as clearly eliminated.^ A subcontract was awarded to Instructional 

Technology Consultants (ITC) (Hood, April 1974). Their proposed evaluation 

study had four major phases: (a) to organize and conduct a one week training 

workshop using the three ufiits to obtain irifoimatioh on attainment of immediate 

objectives; (b) to do a follow-up study to deteimine what uses of the training 

had been made by the^workshop participants when they returned to their jobs;. 

14 ^ 



(c) to determine if and hw the units had been used by- a san^le o'£ tHose ^>Jio had 
^ purchased or otherwise received tlie units, during a^roxiinately the first s'ix 
months they had been on sale; and (d) 'to obtain expert judgments about the . * 
Usefxilness and quality of the three units with particular attention to canpari- 
sons with other training th^ reviewers judgejj to be competitive. ' ^ 

Instructional Technology Consultants carried^-out the study, including the 
development of infomation collection devices, specifications for re\deA>'ers and 
questions for product users, in alrast coirplete independence. Progress was 

f 

reiHewed b>' the program director at the critical dates specified in the proposal. 

' The program director met withUie study director, Dr. Howard Sullivan, on two 

occasions, cnce before the training workshop was conducted, and once after all 

data had been, collected. Tlie data collection instninents were reviewed in draft 

form by the program director, but these reviews focused almost entirely on tech- 

nical measurement or editorial aspects of the instruments. No restrictions^ were 

placed on the content of the instruments. Additioially, there were a nuiribi . of 

telei^one conversations with the study director, but they almost always dealt 

with discussion and elaboration .of aiteinative procedures proposed by ITG on 

' which advice was sou^t before final choices were made. 

I\vo reports on the ^valuation study were prepared (Instructional Technology 

Consultants,, September 1974; December 1974). 'The first report was based on the 

if, 

.information. obtained during the training workshop; the second reported on the 

entire stucfy, including the workshop. The final report and associated exliibits 

are included as an attachment to this report on the development and evaluation 

of the package. Excerpts from the report, cind the five generalizations drawii 

» 

f'V tiic external evaluator at the ^conclusion of the entire study are presented . 
belo^v. The generalizations are taken from 'Ch^q)ter 5 of the final repeat, w)iich 
; included qualifying statements associated with tlie conclusions. Portio^is from 

15 



these qualifyii.g statements are also quoted ivhere tliey appear to suggest 

Lnportant cautions on overgeneralization. ' ^ 

"iV total of 25 participants attended the workshoi-). IVenty-two of 
the participants we^e from school districts in California, ^he 
.remaining three participants were faculty jiveiribers from uniyei^i$ies^ 
in Michigan and Utali ♦ Pretest mean scores, \vhich ranged' from 
32 percent oti the Setting Goals unit to 41 percent on i Malyzing, , 
Objectives , indicate that participants could not perfcm \/ell 
prior to mstructioi on the^s4cilis taught in the units- Following 
instruction, the participants performed modera'tely well (75 percent) ^ 
on the posttest for Setting .Goals .and rather poorly (60 percent 
and 48 percent, respectively) on the posttests for Analyzing Problems 
and Deriving Objectives. Posttest performance on the units could be 
considered to be reasonably s^atisfactory only m Ufiit 1, Setting 
Goals Participant ratings on/ the Iftiit Rating Fom feflect ~ 
favorable attitudes toward each of the three. units. Positive 
ratings outniinb^red negative ratings by a ratio of , more than three 
to one on Uaits 1 and 3 and more than two to one on Ihit 2!" 
(pp. 5-6; 14-15), * 

•TV/elve of the 25 worlcshop participants ^48%) and 10 of the 61 
purch3sers.»(16%)'who returned coirpleted questionnaires had used 
^one or more units with others. TTie users interviewed by telephone 
represented 16 different sites and had used the units with approxi- 
mately 542.sdiool personnel and with IW graduate Stiidents. IMit- 
^related follow-tp activities involving the writing of goals and 
goal indicators or the identification and prioritizing, of problems 
wre being conducted at nine of the 16 sites represented in the 
telephone interviews. Reactions to the units were favorable from 
12 of the 16- sites, and none of the 16 users reported reactions that 
were primarily negative.** (p. 31) 

'•Reviews of the Determining Instructional Purposes mi ts- w*ere^\ • 
obtained fron three professional educators, two of whom have dcme • 
considerable substantive work in the areas covered by the units " 
and a third who had served as a coordinator at the ^^giist workshop 
and v3io represeated a large class of potential iisfers of the units. 
There was considerablef^riation' in the nature bi the comnents of 
tlie three reAaewers and in their reactiqns to the^ units. ^The ^ ^' 
reactions of one reviewer were consistentl)zn^xy favofable,, except 
that he expressed some concern oven the niinber of cofnpe-bing programs 
and the bulk cJf the units. The comments pJE the other ^two reviewers 
were less favorable j even though; each of tiiese reviewers reported 
that he would tecoimiend use of the units for ^particular training 
puTposes associated with his own work. On6 of thfe latter. two 
reviewers generally reacted favorably to ttie Se'ttihg Goals aid 
Analyzing Problems Units and unfavorably toward D privirig Objec^^^ 
Overall, it seems most accurate to describe the reaction% of these 
two revievi^fs tptj^rd the three inits colle.ctively as. beif?g, neutral 
to sl'igji^iy positive. ,All*tliree fe<riewers reported that ^it isivery 
important for school administrators to possess 'the tyf>es'bf skills- 
taught in the units." (p. 41) ^ ' / < 




actions of future users tward the units are lilcely to be favorable . 

The attitudes of the workshpp participants towards the mits was 
Consisten,tly favorable. . .Persons who had used the units witii others ^ ' 

for training or, program development purposes also reported con- 
sistently favorable reactions to the units! Ihe collective reac-*» 
tions of the three reviewej^s^was less favorably overall than those"^ 
of the workshop participants or ^ers^csis who had used the unit% 
with others. However, pne of the reviewers was very positive « }^ 
in hisT evaliiation of the units , and each of the other two asserted 
. that, he wuld recommend their use £ox particular trafhing. purposes ^ 
associated with his own work/' (p. 43j • . ^ 

In most inst ance s, the units are not likely to be used as intended - ; 
by the developers . . • ' . . ' * 

The^manner in vdiifch users proceeded through a unit .also typical ly 
varied from the procedures described in the Cootdinator^s Handbook, 
Only three of the 16 users interviewed by telephone reported thar * 
* ; tliey had used one or more intact units and had closely ^followed ' . . 

the procedures from the Handbook." (pp. 43-44) 

'3. The more general terminal outcome for each unit is the most important ' * 
unit outcome and can be attained without mastering many of the * ^ 
instructional objectives for the unit . • ' 

In the evaluator^s judgment, the general outcome foi^ each unit, 
^ as contrasted with mahy* of *the instructional objectives,, repre- 
sents an outcome that is very important to many, school 'personnel. 
The procedures described in the uni^s, particularly iMits 1 an4 2, , 
are appropriate methods for users to follow an order to attain the 
general outcomes . Users working toward this goal shotild be able 
to refer directly to the procedures in the units .... Thus it seems - 
'apprepriate for "users who want to do so to use only the content ^ ^ 

of the unit that .directly involyes precedures. . .and to by-pass 
' the othe:jr reading material and self -tests. .'..The above general^ 
izati on* . .appears to conflict with statements by all three re-* 
/ ' viewers tliat it is Y^ry important for school administrators to 
possess the types of skills taught in *the units--at least to the 
extent that the reviewers were referring 1:o. the skills represented 
by the ins'tructional objectives.'' (ppi 44-46) 

The present format^ and cost of the units are not conducive to sales . 

. ..multiple copies of the units were purchased for use at only 
one'of the^l6 sites represented by user^ interviewed in this^ 
study. The other individuals who used multiple copies of one . ^ * 

or more units purchased a single copy and reproduced copies 
. fronjit/* (pp. 46-47) ^ , . * 
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"5. If the decision is inade to publish arid disseminate the units , , 
the Laboratory should consider the desirability of >deletixig -the 
Deriving^Objectives unit ! ' \ ^ 

TVro of *the three reviewers expressed rather strong concerns 
, .about the Deriving Objectives unif^^^. One reviewer also note^^ 
th^t more and better cqjipeting products.are^ailable^on the 
topic of developing instruc1;ional objectives than in the 
areas covered by the other units.. Ihe eyaluatof ;^hares the 
perception that the. Dsriviilg Objgctives unit is .not as strong 
- — ' as [the other two units • The part of the ^Deriyiilg Objectives 
unit th&t did consistently receive favorable ccnments from ,the 
reviewers was the programmed text. If the decision^ is made to 
not publish and ^disseminate the unit, consideration could be 
given to marketing the prograimned text as a separate item." 
(pp, 47-48) . . ^ - ^ ^ 

* Conclusion 



9 



It ^eems evident, that the major investment of effort over five years in 



sn^s 

the meticulous develq^ment and testing of a rather novel product resisted in 
the creation of thre^v potentially quite valuable txaining^ units. The particular 
strength ^ the units seems to be in the introduction of participants to pro- 
cesses they can. ^se as they consider, and write program goals and objectives in 
their own schools. The units appear to be relatively ineffective in fostering 
the deVelqpime^t of specific skills, and are probably more verbose than desirable 
for efficient training. One more qiiite major revision of each of ^the three 
units should be^made to, realize fully, the potential valpe of the units, the 
case of Unit 3 -^ Deriving Objectives a con^lete restructuring. of the unit^is 
probably necessary. It seems equally evident that one more revision is needed. 

No funds are presently available to support such a revisiorf, however, a^id 
the. remainder of the interim inventory produced by; th6 Laboratory is relatively 
small.. Tn'is, no further 'plans j^or work on or distribution of .this unit can be 
outlined, other than to distribute the ^; remainder of the inventory in the public 
domain for school personnel to use in such way? as seem appropriate to them. 

\- • - \ 

J ' ^ • ' 
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